Add bpf_redirect_map for AF_XDP sockets#1008
Conversation
| Starting with XDP version 1.1, experimental eBPF support can be enabled by appending an `ADDLOCAL=xdp_ebpf` parameter to the `msiexec.exe` install commands. | ||
| > **Note:** The built-in rules-based program API (`XDP_RULE` / `XDP_MATCH_TYPE` | ||
| > / `XdpCreateProgram`) is deprecated and planned for removal. All users should | ||
| > migrate to eBPF programs. |
There was a problem hiding this comment.
nit: should XDP add support for multi-attach as this is being deprecated?
There was a problem hiding this comment.
Is multi-attach when the program gets loaded once and continuously applies to all interfaces?
Right now, XDP doesn't support that in any program mode - it would be a separate feature request.
There was a problem hiding this comment.
no it means a user can attach multiple different XDP programs to the same interface. Agree that that is outside the scope of this PR and is a separate feature request, but should we file an issue to track that work?
| { | ||
| .header = EBPF_HELPER_FUNCTION_PROTOTYPE_HEADER, | ||
| .helper_id = BPF_FUNC_redirect_map, | ||
| .name = "bpf_redirect_map", |
There was a problem hiding this comment.
since this helper function is also applicable for TC, should we add this in ebpfcore? ebpf supports implementing a "virtual" helper function in ebpfcore that can then be overridden by the extension. that way, multiple program info providers can have their own implementations of the same helper functions?
There was a problem hiding this comment.
Yeah, that would be perfect. Does eBPF-for-Windows support that today? Does a virtual function like that require coordination between extensions, or since TC and XDP programs are different types, they each define their own bpf_redirect_map with no overlap?
Or would this be something like a bpf_redirect_map virtual function gets implemented by eBPF core, and then it calls into a map-specific type helper?
There was a problem hiding this comment.
Yes this is currently supported in ebpf-for-windows. There will be no implementation in ebpfcore, and each extension will need to provide their own implementation. If a program type does not support this helper, and a BPF program of that type uses this helper, load of that program will fail.
See microsoft/ebpf-for-windows#3464 where bpf_get_socket_cookie() was added in the same way.
There was a problem hiding this comment.
Ah, that makes sense. I'll create a similar virtual function and hopefully catch the eBPF 1.2 release.
There was a problem hiding this comment.
Created microsoft/ebpf-for-windows#5302.
# Conflicts: # .github/workflows/ci.yml # docs/usage.md # src/xdp/precomp.h # src/xdp/program.h # src/xdp/xdp.vcxproj # xdp.sln
- Point legacy sample tests (xskfwdkm/xskrestricted/rxfilter/rxfilterperf) at the -deprecated binaries so they validate the deprecated (non-eBPF) samples; the new eBPF samples remain covered by sample-*-test.ps1 + xskmaprx (with eBPF). - Run sample_tests on the self-hosted 1ES Functional pool instead of a GitHub-hosted runner, so the image's pre-configured driver verifier settings match prepare-machine -ForFunctionalTest (fixes 'Reboot required' failure). - Add WPP tracing to XskCanRedirect to capture state/queue at the redirect decision, to diagnose the arm64-only redirect fallback test failures.
The rxfilter/xskfwd/xskrestricted sample tests load native XDP eBPF programs (drop.sys / xsk_redirect.sys), which require XDP's eBPF program-information provider. That provider is only registered when XDP is installed with -EnableEbpf. Without it, program load fails with eBPF error 23 (failed to load program information) and the sample exits with code 1.
xdpbpfexport.exe (run by 'setup.ps1 -Install xdp -EnableEbpf') links EbpfApi.dll, so the eBPF runtime must be installed first. Reorder the sample tests to install eBPF before XDP (and uninstall XDP before eBPF), matching functional.ps1. Fixes xdpbpfexport 0xC0000135 (STATUS_DLL_NOT_FOUND). Validated on hardware: rxfilter sample now processes traffic and passes.
The functional trace profile captures TraceInfo but not TraceVerbose, so the XskCanRedirect diagnostic was not appearing in CI logs. Promote it to TraceInfo and add an EbpfXdpRedirectMap decision summary (key, flags, fallback action, returned action) so the arm64-only redirect fallback failures can be diagnosed from the next CI run's logs.
…k path - Sample tests: uninstall eBPF before XDP so the eBPF redirect program + XSK are torn down before xdp restarts (xskfwd force-kills its loader, leaving the program attached; with XDP uninstalled first the xdp restart hangs). Install order stays eBPF-first for xdpbpfexport. - Redirect diagnostics: only trace XskCanRedirect==FALSE and the RedirectMap fallback path, so high-volume redirect-success traces don't flood the ETL and drop the low-volume fallback-test events needed to diagnose the arm64 failure.
# Conflicts: # tools/common.ps1
With eBPF uninstalled first, the xdp uninstall's xdpbpfexport --clear fails with 0xC0000135 (EbpfApi.dll already gone), leaving xdp loaded and failing the post Check Drivers step. Pass -Force so that failure is downgraded and xdp is fully removed.
The eBPF XSKMAP (bpf_redirect_map to AF_XDP) requires the eBPF for Windows base-map-provider, available in v1.3.0+. functional.ps1 exports the installed runtime version via XDP_EBPF_RUNTIME_VERSION; the eBPF XSKMAP tests skip when it is older than 1.3.0. Adds a SkipTest framework primitive and TEST_SKIP macro.
…irect Use acquire loads for the lockless reads of XSK State and bound RX queue in XskCanRedirect, and a matching release store when teardown transitions State to XskClosing. On weak memory-ordering architectures (arm64) plain loads could observe a stale state and wrongly redirect to a closing socket instead of falling back. Remove the temporary redirect diagnostics.
…fallback The arm64/1.3.0 GenericRxEbpfXskRedirectCloseSocket and QueueMismatch failures are deterministic (Debug and Release), ruling out the memory-ordering race the acquire/release change targeted, so revert it. Both failing tests are the only ones passing a runtime fallback value to bpf_redirect_map (via xsk_redirect_fallback). Add a results map to that eBPF program recording the rx_queue_index and fallback action it computes, and read them back in the test (with a diagnostic warning) to isolate an eBPF-side miscomputation from an XDP-side helper issue.
The arm64 diagnostic confirmed the eBPF program computes the correct rx_queue_index and fallback action, so the drop is XDP-side. Record the value returned by bpf_redirect_map (results_map[2]) and log it, to determine whether the kernel decided XDP_REDIRECT (XskCanRedirect wrongly returned TRUE) or returned the fallback action (drop occurs deeper in the datapath).
The 128-char TAEF warning buffer truncated the combined diagnostic before ProgAction printed, so emit the bpf_redirect_map return action on its own short line. Add low-volume kernel WPP TraceInfo in XskCanRedirect (State/XskQueue/ RxQueue/CanRedirect) and EbpfXdpRedirectMap (Key/Flags/Xsk) to capture the exact redirect-decision inputs for the arm64-only redirect fallback failures. These paths are only exercised by the eBPF XSKMAP redirect tests, so the traces are low volume and retrievable from the functional ETL artifact.
…atapath XdpInvokeEbpf built EBPF_XDP_MD on the stack without initializing ProgTestRunContext. EbpfXdpRedirectMap reads IsProgTestRun = (ProgTestRunContext != NULL); on arm64 the uninitialized stack slot was non-NULL, so real datapath frames were misclassified as prog_test_run, skipping XskCanRedirect and unconditionally returning XDP_REDIRECT. This broke GenericRxEbpfXskRedirectCloseSocket and GenericRxEbpfXskRedirectQueueMismatch on arm64 (deterministic). Initialize ProgTestRunContext = NULL on the datapath. Also removes the temporary diagnostic instrumentation (results_map in xsk_redirect_fallback.c, TEST_WARNING diagnostics in tests.cpp, and kernel TraceInfo in xsk.c/program.c).
| name: All Tests Complete | ||
| if: always() | ||
| needs: [functional_tests, stress_tests, pktfuzz_tests, xskperf_tests, ring_perf_tests, rxfilter_perf_tests, xskfwdkm_test, xskmaprx_test, xskrestricted_test, downlevel_functional_tests] | ||
| needs: [build, build_allpackage, onebranch_build_validation, functional_tests, stress_tests, pktfuzz_tests, xskperf_tests, ring_perf_tests, rxfilter_perf_tests, xskfwdkm_test, xskmaprx_test, xskrestricted_test, downlevel_functional_tests, create_artifacts, sample_tests] |
There was a problem hiding this comment.
Only sample_tests should be added here.
| @@ -1,5 +1,9 @@ | |||
| # XDP_HOOK_ID structure | |||
|
|
|||
| > **Deprecated:** The built-in rules-based program API is planned for removal. | |||
There was a problem hiding this comment.
Revert - hook ID applies more generally to XDP, not just the built-in rules engine.
| ## Extensibility | ||
| - XDP for Windows is designed to be extensible via eBPF (experimental) and a primitive built-in rules-based program engine. | ||
| - XDP for Windows is designed to be extensible via [eBPF programs](./ebpf.md) through integration with [eBPF for Windows](https://github.com/microsoft/ebpf-for-windows). | ||
| - The legacy built-in rules-based program engine is deprecated and planned for removal. See the [eBPF Integration Guide](./ebpf.md) for migration guidance. |
There was a problem hiding this comment.
| - The legacy built-in rules-based program engine is deprecated and planned for removal. See the [eBPF Integration Guide](./ebpf.md) for migration guidance. | |
| - The built-in rules-based program engine is deprecated and planned for removal. See the [eBPF Integration Guide](./ebpf.md) for migration guidance. |
| | `type` | `BPF_MAP_TYPE_XSKMAP` (16) | Identifies this as an XSK redirect map. | | ||
| | `key` | `uint32_t` | The key type. Typically the RX queue index. | | ||
| | `value` | `void *` | Opaque XSK socket handle (populated by user mode). | | ||
| | `max_entries` | Application-defined | Should be >= the number of RX queues you intend to use. | |
There was a problem hiding this comment.
| | `max_entries` | Application-defined | Should be >= the number of RX queues you intend to use. | | |
| | `max_entries` | Application-defined | Should be >= the number of sockets you intend to use. | |
| @@ -0,0 +1,326 @@ | |||
| # eBPF Redirect Map (XSKMAP) | |||
There was a problem hiding this comment.
This entire document should be careful to frame the bpf_redirect_map function as conceptually supporting / extensible to other map types; the XSKMAP is the only currently supported type.
| ```c | ||
| struct { | ||
| __uint(type, BPF_MAP_TYPE_XSKMAP); | ||
| __type(key, uint32_t); |
There was a problem hiding this comment.
Should be a 64-bit key (needs to be updated across the entire PR).
| AF_XDP socket found in the map. | ||
| - **Fallback action** -- on failure (extracted from the low two bits of `flags`). Failure can occur when: | ||
| - The key does not exist in the map (no XSK handle for that queue). | ||
| - The XSK socket is not in a valid state for redirect (e.g., not yet |
There was a problem hiding this comment.
Also include mismatched queue as a documented invalid state.
| - The XSK socket is not in a valid state for redirect (e.g., not yet | ||
| activated or already closing). | ||
|
|
||
| ## Examples |
There was a problem hiding this comment.
We should also link to the concrete sample source code in this repo.
| | Project | Role | | ||
| |---------|------| | ||
| | **[XDP for Windows](https://github.com/microsoft/xdp-for-windows)** | Provides the high-performance data path: NIC hook points, AF_XDP sockets, shared-memory rings, and the XDP driver (`xdp.sys`). Registers an XDP program type and helper functions with the eBPF runtime. | | ||
| | **[eBPF for Windows](https://github.com/microsoft/ebpf-for-windows)** | Provides the eBPF runtime: program verification (PREVAIL verifier), JIT compilation (uBPF), map infrastructure, and the NMR-based extension model that XDP plugs into. | |
There was a problem hiding this comment.
Omit references to JIT; the released versions of eBPF for Windows do not support JIT.
| | **[XDP for Windows](https://github.com/microsoft/xdp-for-windows)** | Provides the high-performance data path: NIC hook points, AF_XDP sockets, shared-memory rings, and the XDP driver (`xdp.sys`). Registers an XDP program type and helper functions with the eBPF runtime. | | ||
| | **[eBPF for Windows](https://github.com/microsoft/ebpf-for-windows)** | Provides the eBPF runtime: program verification (PREVAIL verifier), JIT compilation (uBPF), map infrastructure, and the NMR-based extension model that XDP plugs into. | | ||
|
|
||
| An eBPF program compiled to BPF bytecode is loaded by the eBPF for Windows |
There was a problem hiding this comment.
Not true - only the native offline-compiled .sys file mode is supported officially today.
| xdp-setup.ps1 -Install xdpebpf | ||
| ``` | ||
|
|
||
| ### MSI (<= v1.2 only) |
There was a problem hiding this comment.
Let's leave MSI undocumented here.
| See [eBPF Redirect Map (XSKMAP)](ebpf-redirect-map.md) for a comprehensive | ||
| guide. | ||
|
|
||
| ### `bpf_xdp_adjust_head` |
There was a problem hiding this comment.
Not actually implemented - do not document here.
| / `XDP_RULE_ACTION`) is deprecated and planned for removal. All users should | ||
| migrate to eBPF programs. | ||
|
|
||
| ### Migration Quick Reference |
There was a problem hiding this comment.
remove this section
| | `XDP_MATCH_IPV4_DST_MASK` | Parse IPv4 header, apply bitmask to `dst_addr` | | ||
| | Port set matching | Use a `BPF_MAP_TYPE_HASH` or `BPF_MAP_TYPE_ARRAY` map for port lookups | | ||
|
|
||
| ### Example: Migrating a UDP Port Filter |
There was a problem hiding this comment.
Remove this section
| We do not officially support eBPF programs yet. We are [integrating](https://github.com/microsoft/xdp-for-windows/issues/7) with the | ||
| [eBPF for Windows project](https://github.com/microsoft/ebpf-for-windows), which is also in a pre-release stage. In the meantime, we | ||
| have built a barebones [program](/published/external/xdp/program.h) module. | ||
| Yes. XDP supports eBPF programs through [integration](https://github.com/microsoft/xdp-for-windows/issues/7) with the |
There was a problem hiding this comment.
Clarify the version where support was added (tentatively v1.4)
| return XDP_PASS; | ||
| } | ||
| protocol = ip->Protocol; | ||
| transport = (char *)ip + (ip->HeaderLength * 4); |
There was a problem hiding this comment.
IDK if we want to implicitly allow invalid header lengths here - for simplicity, maybe it's fine, but we should add a comment explicitly not validating it.
| *MapContext = NULL; | ||
|
|
||
| if (MapType != BPF_MAP_TYPE_XSKMAP) { | ||
| return EBPF_INVALID_ARGUMENT; |
There was a problem hiding this comment.
Let's do EBPF_OPERATION_NOT_SUPPORTED instead.
|
|
||
| if (!IsEqualGUID(ProgramType, &ExpectedProgramType)) { | ||
| TraceError(TRACE_CORE, "XSKMAP only supports XDP program type"); | ||
| return EBPF_OPERATION_NOT_SUPPORTED; |
There was a problem hiding this comment.
Use single point of return.
| // | ||
| Context = Binding->ClientDispatch.epoch_allocate_with_tag(sizeof(*Context), XDP_POOLTAG_MAP); | ||
| if (Context == NULL) { | ||
| return EBPF_NO_MEMORY; |
There was a problem hiding this comment.
Use single point of return.
| // | ||
| // Provider dispatch table for XSKMAP operations. | ||
| // | ||
| static ebpf_base_map_provider_dispatch_table_t XdpXskmapProviderDispatchTable = { |
There was a problem hiding this comment.
should be const, too
There was a problem hiding this comment.
same for others below.
| min(sizeof(Binding->ClientDispatch), ClientData->base_client_table->header.total_size)); | ||
|
|
||
| WriteULong64NoFence( | ||
| (volatile UINT64 *)&XdpXskmapContextOffset, ClientData->map_context_offset); |
There was a problem hiding this comment.
No need for the volatile keyword in cast
| XDP_XSKMAP_CONTEXT *Context; | ||
|
|
||
| Context = *(XDP_XSKMAP_CONTEXT **)MAP_CONTEXT(Map, ReadULong64NoFence(&XdpXskmapContextOffset)); | ||
| if (Context == NULL) { |
There was a problem hiding this comment.
We should ensure we have test coverage of non-XSKMAPs passed to the redirect helper, either in functional or spin tests, or ideally both.
| // | ||
| ASSERT(!(Flags & EBPF_MAP_OPERATION_HELPER)); | ||
|
|
||
| if (InValueSize != sizeof(HANDLE) || InValue == NULL) { |
There was a problem hiding this comment.
Spin test must cover incorrect key/value sizes and values.
| // | ||
| static | ||
| bool | ||
| EbpfXskmapSupported() |
There was a problem hiding this comment.
Let's split a sub-routine out that retrieves the major, minor, patch version of eBPF from the env var, and this function's business logic becomes simple.
| return (Major > 1) || (Major == 1 && Minor >= 3); | ||
| } | ||
|
|
||
| #define SKIP_IF_EBPF_XSKMAP_UNSUPPORTED() \ |
There was a problem hiding this comment.
Don't create a macro like this - just add the same 3 lines everywhere.
|
|
||
| #define SKIP_IF_EBPF_XSKMAP_UNSUPPORTED() \ | ||
| if (!EbpfXskmapSupported()) { \ | ||
| TEST_SKIP("eBPF XSKMAP requires eBPF for Windows v1.3.0 or later"); \ |
There was a problem hiding this comment.
Don't hard-coded 1.3.0 here - that belongs to the map supported function. Just print eBPF does'nt support
| ::GenericRxEbpfXskMapControlPath(); | ||
| } | ||
|
|
||
| TEST_METHOD_PRERELEASE(GenericRxEbpfXskRedirectCloseSocket) { |
There was a problem hiding this comment.
Should not be the prerelease variant - this will be officially supported
There was a problem hiding this comment.
(Same for all new tests)
| EnterCriticalSection(&sharedEbpfProgram.Lock); | ||
| if (sharedEbpfProgram.BpfObject != NULL && sharedEbpfProgram.XskMapFd >= 0) { | ||
| MapFd = sharedEbpfProgram.XskMapFd; | ||
| InterlockedIncrement(&sharedEbpfProgram.RefCount); |
There was a problem hiding this comment.
Let's use the XDP refcount type, if its header is available here.
| // | ||
| // Store the eBPF program and map FD in the shared interface-level set. | ||
| // | ||
| { |
There was a problem hiding this comment.
Do not create gratuitous scopes in C.
| } | ||
| } | ||
|
|
||
| $env:XDP_EBPF_RUNTIME_VERSION = if ([string]::IsNullOrEmpty($EbpfVersion)) { Get-EbpfVersion } else { $EbpfVersion } |
There was a problem hiding this comment.
Split this into multiple lines, no = if pattern.
|
|
||
| for ($i = 0; $i -lt $QueueCount; $i++) { | ||
| Start-Process $ArtifactsDir\test\rxfilter.exe -ArgumentList ` | ||
| Start-Process $ArtifactsDir\test\rxfilter-deprecated.exe -ArgumentList ` |
There was a problem hiding this comment.
If this script is invoking the deprecated binary, it itself should be labeled deprecated. However, it would be better if this wrapper script can conditionally invoke the deprecated version, otherwise invoke the current eBPF version. Same for the rest of the helper scripts.
Description
Describe the purpose of and changes within this Pull Request.
WIP
Resolves #941.
Testing
Do any existing tests cover this change? Are new tests needed?
CI, automated functional and spin tests added.
Documentation
Is there any documentation impact for this change?
Yes, docs added, and also deprecation notices are added for the built-in program API, now that the feature gap is effectively closed.
Installation
Is there any installer impact for this change?
N/A.